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EXECUTIVE  SUMMARY 


This  effort  was  initiated  to  evaluate  the  effectiveness  of  a  commercial  product  that  generates 
speech  from  written  text,  in  the  presentation  of  Airport  Terminal  Information  Service  (ATIS) 
messages.  The  Federal  Aviation  Administration  (FAA)  plans  to  deploy  workstations  for  airport 
tower  operation  that  will  generate  relevant  air  traffic  control  messages,  including  ATIS, 
transmission  via  Data  Link.  However,  for  those  aircraft  not  equipped  with  Data  Link,  such  as 
general  aviation,  the  voice  broadcast  of  ATIS  must  be  maintained.  Text  to  speech,  resident  in  the 
workstations,  is  envisioned  as  a  means  to  provide  this  capability. 

This  evaluation  was  concentrated  on  general  aviation  operations  and  consisted  of  two  flights  from 
a  point  west  of  Woodstown,  NJ,  to  an  approach  and  landing  at  Atlantic  City  International  Airport, 
NJ.  Each  flight  was  conducted  in  a  simulator  of  a  Cessna  421  twin  aircraft,  and  airspace 
conditions  included  other  traffic,  light  turbulence,  and  deteriorating  weather.  During  the  flight, 
ATIS  weis  updated,  and  pilot  subjects  were  required  to  listen  to  each  message.  Four  conditions  of 
flight  scenarios  and  ATIS  messages  made  up  the  flight  experience.  One  flight  scenario  was 
routine,  with  background  traffic  equivalent  to  an  average  day  in  the  Atlantic  City  area.  The  other 
flight  scenario  was  designed  to  be  higher  workload,  with  more  background  radio  traffic,  and 
involved  a  runway  change  due  to  weather.  An  in-flight  observer  noted  pilot  awareness  of  any 
changes  in  the  messages,  as  well  as  overall  message  content.  Pilot  subject  commentary  was  also 
noted  by  the  observer.  Across  all  scenarios,  natural  speech  ATIS  was  contrasted  with  the 
synthesized  ATIS. 

Data  collection  included  subjective  assessments  by  the  pilot  subjects  via  commentary  and  written 
questionnaires,  and  objective  data  such  as  the  number  of  repetitions  that  pilot  subjects  listen  to  the 
ATIS. 

Ten  pilot  subjects  from  the  Atlantic  Area  participated  in  the  study.  Overall,  their  subjective 
ratings  were  favorable;  indicated  by  a  rating  of  5.6  on  a  scale  of  1  to  7.  Moreover,  no  objective 
data  suggested  that  there  was  any  loss  of  ATIS  information  due  to  the  synthetic  speech  compared 
to  the  natural  speech  case.  Data  did  suggest  that  longer  multisyllablic  words,  such  as  would  be 
found  in  NOT AMS  (Notices  to  Airmen)  would  be  problematic,  and  cause  additional  workload  for 
pilots.  Shorter  information  bits  such  as  altimeter  settings  and  winds  were  not  demonstrated  to  be 
a  problem  by  the  data. 

This  study  showed  that  synthetic  text  to  speech  as  implemented  was  rated  favorable,  but  that 
further  work  is  needed  prior  to  field  deployment.  Due  to  the  rigor  of  the  experimental  design 
used  in  this  study,  it  can  also  form  the  basis  for  an  evaluation  of  an  improved  text  to  speech 
system.  Ratings  used  in  this  study  can  be  applied  in  a  future  evaluation,  and  comparisons  can  be 
made. 


1.  OVERVIEW. 


1.1  BACKGROUND  AND  OBJECTIVES. 

Air  traffic  controllers  currently  broadcast  Automatic  Terminal  Information  Service  (ATIS) 
messages  verbally.  The  present  system  sometimes  suffers  from  poor  intelligibility  of  the 
broadcase  due  to  inconsistent  recording  quality,  differences  in  voice  characteristics,  and  noise  on 
the  radio  channel.  One  proposed  enhancement  is  the  use  of  digitized/  synthesized  speech  to 
replace  the  recorded  human  voice  to  gain  consistency  in  the  broadcast  message. 

Moreover,  a  need  exists  to  retain  both  digital  and  voice  messages  for  the  mixed  environment  of 
aircraft  that  are  Data  Link  equipped  and  for  aircraft  that  do  not  have  digital  capability.  However, 
providing  both  types  of  messages  effectively  doubles  the  time  required  for  air  traffic  controllers  to 
prepare  the  messages.  A  potential  solution  is  to  use  voice  synthesis  to  "read"  the  digital  messages 
and  transmit  them  in  synthetic  voice. 

Speech  synthesis  of  ATIS  messages  has  not  been  fully  tested  in  an  operational  environment.  The 
Federal  Aviation  Administration  (FAA)  Technical  Center,  Atlantic  City  International  Airport, 
New  Jersey,  under  contract  DTFA03-94-Q-00022,  tasked  ARINC  and  CTA  Incorporated  to 
evaluate  the  effectiveness  of  synthetic  voice  ATIS  messages  in  a  simulated  environment.  The 
evaluation,  consisted  of  a  flight  demonstration  in  the  General  Aviation  Trainer  (GAT),  flight 
simulator  at  the  FAA  Technical  Center.  The  GAT  is  a  motion-based  general  aviation  trainer 
representing  a  Cessna  421  aircraft  and  has  out-of-the- window  visual  renditions  of  the  Atlantic 
City  International  Airport  airport  and  environs. 

The  demonstration  was  held  from  November  28  through  December  2, 1994.  Pilots  flew  two 
scenarios  and  heard  two  ATIS  messages  during  each  scenario — one  using  human  voice  and  one 
using  synthetic  voice. 

The  demonstration  had  the  following  objectives: 

a.  Evaluate  the  synthetic  voice  messages  using  measures  of  speed,  clarity  and 
intelhgibihty,  and  overall  quality. 

b.  Compare  the  pilots'  responses  to  the  synthetic  voice  messages  with  their  responses  to 
the  human  voice  messages. 

These  objectives  were  achieved  by  evaluating  subjective  and  objective  measures  collected  from 
the  pilots  and  cockpit  observers  during  the  flight  demonstration. 

1.2  ORGANIZATION  OF  REPORT. 

Section  2  describes  the  methodology  used  for  the  evaluation.  Section  3  describes  the  results,  and 
section  4  presents  conclusions  and  recommendations. 
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2.  METHODOT.OrTY. 


Ten  pilots  from  the  Atlantic  City  area  were  selected  for  the  demonstration.  All  10  had  some 
previous  experience  with  AXIS.  For  the  purposes  of  this  study,  pilots  with  more  than  300  hours 
of  multiengine  experience  were  considered  expert,  while  those  with  less  than  300  hours  were 
considered  novices.  Of  the  initial  10  pilots  selected,  5  were  novices  and  5  were  experts. 
However,  two  of  the  expert  pilots  could  not  complete  the  demonstration  because  of  conflicting 
schedules  and  both  of  the  available  replacement  pilots  were  novice  pilots.  Therefore,  the 
demonstration  was  conducted  with  seven  novice  pilots  and  three  expert  pilots. 

Two  flight  scenarios  were  designed  to  provide  the  pilots  with  different  flight  experiences  to 
reduce  the  amount  of  learning  resulting  from  repetition.  Each  scenario  was  developed  based  on 
an  approach  to  Atlantic  City  International  Airport,  lasted  about  40  minutes,  and  included  some 
turbulence  and  background  noise  for  realism.  The  scenarios  presented  the  following  routes: 

a.  Scenario  1;  CANNY-MTV-SIE-ACY  via  VHF  omnirange  (VOR),  runway  31 

b.  Scenario  2:  CANNY-DONIL-VCN-ACY  via  instrument  landing  system  (ILS), 
runway  13 

Appendix  A  illustrates  both  flight  routes. 

Four  different  ATIS  messages  were  developed  for  the  demonstration,  and  each  was  recorded 
twice — once  using  human  voice  and  once  using  synthetic  voice.  All  of  the  messages,  which  were 
developed  with  the  help  of  a  certified  terminal  controller,  contained  information  about  conditions 
such  as  weather,  wind,  altimeter,  runway,  and  taxiways.  In  addition,  the  test  messages  used 
excerpts  from  actual  ATIS  broadcasts  from  the  airport.  The  controller's  voice  was  used  for  the 
human  voice  message,  and  a  Digital  Equipment  Corporation  DECtalk  System  was  used  to 
generate  the  synthetic  voice  message.  Although  DECtalk  allows  for  customization  of  words  and 
phrases,  that  option  was  not  employed.  The  basic  text  to  speech  synthesis  inherent  in  the  system 
was  used.  Both  the  human  and  synthetic  voice  messages  were  recorded  on  a  SiUcon  Graphics 
workstation  and  played  over  the  simulator  audio  system  during  each  flight.  Appendix  B  presents 
the  contents  of  the  four  ATIS  messages. 

Two  sessions  were  conducted  each  day.  During  the  first  hour  of  a  session,  the  pilot  subject 
received  training  in  the  use  of  the  simulator  and  background  information  about  the  simulation. 

Following  a  break,  one  of  the  flight  scenarios  was  run  for  data  collection.  Two  of  the  four 
messages  were  played  during  the  scenario-one  in  synthetic  voice  and  one  in  human  voice.  After 
another  break,  the  second  flight  scenario  was  run,  during  which  the  remaining  two  messages  were 
played — one  in  synthetic  voice  and  one  in  human  voice.  The  order  of  both  the  messages  and  type 
of  voice  used  were  varied  throughout  the  data  collection.  For  half  of  the  simulations,  a  human 
voice  message  was  presented  first,  followed  by  a  synthetic  voice  message;  the  other  half  of  the 
simulations  followed  the  reverse  order.  Each  pilot  flew  both  scenarios.  Half  of  the  pilots 
experienced  scenario  1  first,  and  half  experienced  scenario  2  first.  Appendix  C  lists  the  order  of 
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presentation  for  each  pilot,  and  appendix  D  presents  the  daily  schedule.  The  pilots  flew  under 
visual  flight  rules  (VFR)  conditions  and  could  use  autopilot  at  their  discretion.  Pilots  were 
requested  to  check  their  AXIS  messages  twice  during  the  simulation — once  just  after  starting  the 
flight  and  again  just  before  contacting  the  tower  during  approach. 

A  cockpit  observer,  who  was  also  a  pilot,  sat  in  the  copilot’s  seat  during  each  session.  After  an 
AXIS  message  was  received,  the  observer  recorded  the  number  of  times  the  pilot  listened  to  each 
message  and  recorded  whether  the  pilot  took  any  of  the  following  actions  after  hearing  a  message: 

a.  Changed  the  altimeter  setting. 

b.  Altered  the  approach  based  on  the  change-of-mnway  information. 

c.  Correctly  recorded,  on  the  pilot's  knee  pad,  information  that  was  changed  from 
the  previous  message. 

d.  Correctly  repeated  the  altimeter  setting  or  runway  change  when  questioned  by 
the  observer. 

These  observations  and  pilot's  comments  were  recorded  on  a  questionnaire  presented  in  appendix 
E.  The  observer  was  also  encouraged  to  record  his  or  her  own  opinions  and  comments  about  the 
quality  of  the  synthetic  voice  messages. 

To  create  realistic  background  traffic  and  communications,  an  air  traffic  controller  provided  air 
traffic  control  (ATC)  instructions  to  the  pilot  subjects  and  provided  cross  talk  with  three 
additional  pseudo  pilots  who  acted  as  other  aircraft  to  provide  simulated  traffic. 

Following  each  session,  the  pilot  completed  a  questionnaire,  rating  the  speed,  clarity  and 
intelligibility,  and  overall  quality  of  the  synthetic  voice  message.  The  questionnaire  also  asked 
whether  the  pilot  preferred  natural  or  synthetic  voice  and  solicited  any  additional  comments  the 
pilot  wished  to  make.  Appendix  F  presents  a  copy  of  the  questionnaire,  and  appendix  G  lists  the 
pilots'  ratings  and  comments. 

3.  RESULTS. 

Table  1  summarizes  the  questionnaire  responses  of  the  seven  novice  and  three  experienced  pilots 
who  participated  in  this  demonstration.  The  results  are  discussed  in  the  following  subsections. 

3.1  HOURS  OF  EXPERIENCE. 

The  mean  hours  of  experience  was  43  for  the  seven  novice  pilots  and  1,433  for  the  three 
experienced  pilots.  One  of  the  novice  pilots  reported  having  only  single-engine  experience. 
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3.2  SPEED. 


Pilots  were  asked  to  rate,  on  a  7-point  scale,  the  speed  of  the  synthetic  voice,  where  1  was  too 
fast,  4  was  adequate,  and  7  was  too  slow.  The  overall  mean  was  4.3  (standard  deviation  of 
0.66) — only  slightly  slower  than  adequate.  This  indicates  that  no  significant  changes  are  required 
to  the  speed  of  the  synthetic  voice.  Three  pilots  (all  novices)  commented  that  the  message  was 
too  slow  and  rated  the  speed  as  6,  5,  and  5,  respectively.  All  of  the  other  pilots  rated  the  speed  of 
the  synthetic  voice  as  4. 

3.3  CLARITY  AND  INTELLIGroiLITY. 

Pilots  were  asked  to  rate,  on  a  7-point  scale,  the  clarity  and  intelligibility  of  the  synthetic  voice, 
where  1  was  not  clear,  4  was  somewhat  clear,  and  7  was  completely  clear.  The  overall  mean  was 
5.6  (standard  deviation  of  1.36).  None  of  the  pilots  rated  clarity  and  intelligibility  as  less  than  4, 
and  four  of  the  pilots  rated  it  as  7.  For  the  six  pilots  who  rated  clarity  and  intelligibility  as  less 
than  7,  some  improvement  was  desired.  Some  commented  that  they  had  difficulty  with  certain 
words  and  others  indicated  that  they  thought  the  voice  had  a  foreign  accent. 

TABLE  1.  PILOT  RATINGS  OF  SYNTHETIC  VOICE  ATIS  MESSAGES 


Pilot 

Hours  of 
Experience 

Speed 

Clarity 

Quality 

Preference 

II  Novice  Pilots  || 

1 

92 

6.00 

2.00 

Natural 

2 

75 

4.00 

6.00 

Natural 

3 

30 

4.00 

4.00 

Natural 

4 

32 

4.00 

7.00 

No  preference 

5 

40 

5.00 

5.00 

Natural 

7 

0 

5.00 

7.00 

No  preference 

10 

30 

4.00 

7.00 

Synthetic 

Mean 

43 

4.57 

6.00 

5.43 

1  Experienced  Pilots  || 

6 

4.00 

Natural* 

8 

HH 

6.00 

Synthetic 

9 

■EH 

4.00 

Natural 

Mean 

1,433 

4.67 

4.67 

Total 

4.30 

5.20 

Mean 

Standard 

Deviation 

0.66 

1.36 

1.16 

*  This  was  the  only  pilot  who  missed  the  change  of  altimeter  settings. 
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It  might  be  expected  that  more  experienced  pilots,  who  have  heard  more  AXIS  messages  given  by 
a  greater  diversity  of  controllers,  would  be  able  to  understand  the  synthetic  voice  more  readily 
than  less  experienced  pilots.  Such  was  not  the  case.  The  mean  rating  from  novice  pilots  was  6.0 
and  from  experienced  pilots  was  4.67.  Examination  of  the  comments  did  not  explain  this  disparity. 

3.4  OVERALL  QUALITY. 


Pilots  were  asked  to  rate,  on  a  7-point  scale,  the  overall  quality  of  the  synthetic  voice,  where  1 
was  not  at  all  acceptable  and  7  was  completely  acceptable.  The  responses  varied  greatly.  Three 
pilots  rated  the  overall  quality  as  7,  two  rated  it  as  6,  one  rated  it  as  5,  three  rated  it  as  4,  and  one 
rated  it  as  2.  The  overall  mean  was  5.2  (standard  deviation  of  1.16).  These  ratings  indicate  that 
although  the  overall  quality  of  the  synthetic  voice  was  acceptable,  there  is  a  need  for 
improvement.  The  comments  from  the  pilots  indicated  the  following  problems: 

a.  Synthetic  voice  is  difficult  to  interpret  in  high-workload  situations. 

b.  Uninitiated  pilots  need  time  to  become  accustomed  to  the  system. 

c.  Clarity  needs  to  be  improved. 


d.  Speed  should  be  increased. 

e.  The  foreign-sounding  accent  hinders  understandability. 

f.  More  attention  and  concentration  are  required  than  for  human  voice. 

Novice  pilots  rated  the  acceptability  of  the  quality  slightly  higher  (mean  of  5.43)  than  did 
experienced  pilots  (mean  of  4.67). 

3.5  PREFERENCE. 


The  pilots  were  asked  which  type  of  AXIS  message  they  preferred — natural  voice,  synthetic 
voice,  or  no  preference.  Six  of  the  pilots  indicated  a  preference  for  natural  voice,  two  preferred 
synthetic  voice,  and  two  have  no  preference.  Although  all  of  the  pilots  rated  clarity  and 
intelligibility  as  at  least  somewhat  clear,  and  all  but  one  pilot  considered  overall  quality  to  be  at 
least  moderately  acceptable,  over  half  preferred  natural  voice. 

It  is  important  to  note  that  although  the  pilots  had  little  or  no  previous  experience  with  synthetic 
voice,  4  of  the  10  either  had  no  preference  or  preferred  the  synthetic  voice.  The  question  can  be 
raised  as  to  what  impact  experience  with  synthetic  voice  has  on  preference  and  acceptability. 

Flight  experience  did  not  present  a  correlation  with  preference.  Of  the  novice  pilots,  four 
preferred  natural  voice,  one  preferred  synthetic  voice,  and  two  had  no  preference.  Of  the  three 
experienced  pilots,  two  preferred  natural  voice,  and  one  preferred  synthetic  voice. 
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3.6  BACKGROUND  TRAFFIC 


After  the  first  trial,  an  additional  item  was  handwritten  into  the  questionnaire.  Pilots  were  asked  to 
rate  the  background  traffic  provided  by  the  simulation  environment.  Of  the  nine  pilots  who 
responded,  six  rated  traffic  as  normal,  and  three  rated  it  as  normal  to  light.  Some  of  the  pilots 
commented  that  background  traffic  was  normal  for  Atlantic  City  but  light  for  other  areas.  None 
of  the  pilots  rated  the  traffic  as  heavy. 

3.7  OBJECTIVE  MR  AST  TRRS 

All  of  the  pilots  correctly  altered  their  approach  based  on  change-of-runway  information  given  in 
the  AXIS  messages.  This  reaction  occurred  regardless  of  whether  the  pilot  heard  the  human 
voice  or  synthetic  voice  message. 

Only  one  of  the  pilots  (pilot  #6)  missed  the  change  of  altimeter  reading.  This  occurred  for  both 
synthetic  voice  messages.  The  observer  noted  that  this  pilot  had  difficulty  handling  the  workload 
and  missed  the  change  of  altimeter  reading  for  the  human  voice  message  as  well.  It  is  interesting 
to  note  that  although  this  pilot  missed  instructions,  he  rated  the  background  traffic  as  normal  to 

light  and  rated  the  synthetic  voice  as  4  for  speed,  4  for  clarity  and  intelligibility,  and  4  for  overall 
quality. 


Table  2  presents  the  results  from  the  Observer  Questionnaires  and  shows  for  each  pilot,  the  order 
of  the  scenarios,  the  mode  of  presentation  of  each  of  the  messages  and  the  number  of  times  the 
pilot  listened  to  each  message  (repetitions).  Table  3  shows  the  mean  number  of  times  that  the 
pilots  listened  to  a  message.  The  mean  number  of  times  pilot  subjects  listened  to  AXIS  was  1 .27 
for  both  natural  and  synthetic  speech  cases.  One  pilot  listened  to  two  human  voice  messages 
three  times. 

The  Charlie  and  Echo  messages  were  always  given  early  in  a  scenario  and  represented  times  of 
relatively  low  workload.  The  Delta  and  Foxtrot  messages  were  given  later  in  a  scenario  and 
represented  times  of  higher  workload.  If  higher  workload  increased  the  number  of  times  the 
synthetic  messages  were  listened  to,  the  Delta  and  Foxtrot  messages  should  have  required  more 
repetitions.  However,  the  synthetic  voice  Delta  message  was  listened  to  an  average  of  1.5  times 
(compared  with  1.25  times  for  human  voice),  and  the  synthetic  voice  Foxtrot  message  was 
listened  to  an  average  of  1.0  times  (compared  with  1.5  times  for  human  voice).  Therefore,  no 
connection  was  demonstrated  between  workload  and  the  number  of  times  that  a  pilot  listened  to 
the  synthetic  voice  messages. 
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TABLE  2.  NUMBER  OF  REPETITIONS  OF  ATIS  MESSAGES  BY  SCENARIO, 
SOURCE,  MESSAGE  AND  PILOT  NUMBER 


Pilot  Number 

Scenario 

Source/Message 

Repetitions 

1 

1 

Human/Charlie 

3 

DECtalk/Delta 

2 

2 

DECtalk/Echo 

2 

Human/Foxtrot 

3 

2 

2 

Human/Echo 

1 

DECtalk/Foxtrot 

1 

1 

DECtalk/Charlie 

2 

Human/Delta 

2 

3 

2 

DECtalk/Echo 

2 

Human/Foxtrot 

2 

1 

Human/Charlie 

1 

DECtaUc/Delta 

2 

4 

1 

Human/Charlie 

1 

DECtalk/Delta 

1 

2 

DECtalk/Echo 

1 

Human/Foxtrot 

1 

5 

1 

Human/Charlie 

1 

DECtalk/Delta 

1 

2 

DECtalk/Echo 

1 

Human/Foxtrot 

1 

6 

1 

Human/Charlie 

1 

DECtalk/Delta 

1 

2 

DECtalk/Echo 

1 

Human/Foxtrot 

1 

7 

2 

DECtalk/Echo 

1 

Human/Foxtrot 

1 

1 

Human/Charlie 

1 

DECtalk/Delta 

2 

8 

2 

Human/Echo 

1 

DECtalk/Foxtrot 

1 

1 

DECtalk/Charlie 

1 

Human/Delta 

1 

9 

1 

DECtalk/Charlie 

1 

Human/Delta 

1 

2 

Human/Echo 

1 

DECtalk/Foxtrot 

1 

10 

2 

Human/Echo 

1 

DECtalk/Foxtrot 

1 

1 

DECtalk/Charlie 

1 

Human/Delta 

1 

Table  3  shows  the  mean  number  of  repetitions  for  each  message  and  source. 

TABLE  3.  MEAN  REPETITIONS  OF  ATIS  MESSAGES  BY  SOURCE  AND  MESSAGE 


Source 

Message 

No.  Pilots 

Mean  Repetitions 

Human 

Charhe 

6 

1.33 

Human 

Delta 

4 

1.25 

Human 

Echo 

4 

1.0 

Human 

Foxtrot 

6 

1.50 

Mean  -  Human 

1.27 

DECtalk 

Charlie 

4 

1.25 

DECtalk 

Delta 

6 

1.50 

DECtalk 

Echo 

6 

1.33 

DECtalk 

Foxtrot 

4 

1.00 

Mean  - 
DECtalk 

1.27 

4,  DATA  ANALYSTS. 

The  objective  data  shows  that  the  pilot  subjects  received  sufficient  information  from  namral  or 
synthetic  ATIS  to  execute  an  approach  and  landing  at  Atlantic  City.  The  subject's  ability  to 
acquire  change  of  runway  and  altimeter  information  from  ATIS  was  consistent.  It  is  noted  that 
these  two  pieces  of  information  are  short  and  familiar  to  pilots.  In  the  subjective  data  for  the 
synthetic  case,  pilots  and  observers  commented  that  (1)  longer  multisyllablic  words,  and 
(2)  NOT  AMS  may  be  more  difficult  to  interpret  from  synthetic  speech.  Combining  these 
subjective  and  objective  results,  it  becomes  evident  that  shorter  length  message  segments  that 
contain  familiar,  or  at  least  expected  information  were  more  readily  understood  by  the  pilot 
subjects  than  the  longer  more  complex  segments.  Longer,  more  complex  messages  required  more 
concentration  by  the  pilot  subjects,  to  acquire  all  of  the  information  contained. 

Workload  was  reported  as  a  concern  by  several  of  the  pilot  subjects  in  the  test  scenarios,  where 
workload  associated  with  the  flying  task  was  designed  to  be  medium.  It  must  be  assumed 
therefore,  that  synthetic  ATIS,  in  the  configuration  used  for  this  demonstration,  may  be 
unacceptable  in  flights  where  the  workload  is  high. 

5.  CONCLUSIONS  AND  RECOMMENDATIONS 

Both  experienced  and  novice  pilots  favorably  rated  intelligibility  and  overall  quality.  Although 
some  pilots  indicated  a  belief  that  the  synthetic  voice  messages  added  to  their  workload,  they  did 
not  need  to  listen  to  additional  iterations  of  the  messages.  Furthermore,  only  one  pilot  missed 
important  information  from  the  synthetic  message  and  that  same  pilot  missed  important 
information  given  by  the  human  controller  as  well. 
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Perhaps  the  most  significant  result  of  this  study  is  a  set  of  ratings  of  a  typical  medium  quality 
speech  synthesis  system.  This  study  demonstration  was  conducted  with  sufficient  rigor  to  provide 
repeatable  measures  in  future  evaluations.  Two  possible  means  exist  for  improvements  ot  the 
speech  synthesis  process:  (a)  acquire  a  more  sophisticated  system,  and  (b)  customize  the  more 
difficult  words  within  the  DECtalk  vocabulary  to  provide  more  natural  speech  overall. 

The  first  alternative  is  technology  (and  expense)  driven,  although  there  is  a  very  promising  system 
available,  called  Eloquent  Technology,  which  has  been  introduced  in  the  last  several  months.  The 
second  alternative  uses  existing  technology  and  is  feasible.  The  customization  of  the  vocabulary 
would  be  equivalent  to  a  site  adaptation  exercise,  and  would  be  performed  within  the  region. 

Follow-on  studies  in  the  GAT  are  suggested  because  performance  gains  over  the  current  synthesis 
system  can  be  assessed  using  similiar  scenarios  and  data  collection  methods.  Furthermore,  if  an 
additional  study  is  performed  using  customized  DECtalk,  the  amount  of  effort  spent  improving 
the  vocabulary  will  be  documented  as  a  case  study  of  site  adaptation.  From  this  data,  the  cost  of 
adaptations  at  other  airports  may  be  predicted. 

Lastly,  if  improvements  to  the  base  technology  are  realized,  these  systems  may  also  be 
comparatively  evaluated  in  a  GAT  study. 

It  is  therefore  recommended  that  the  Technical  Center  plan  and  conduct  at  least  one  additional 
study,  with  natural  voice  as  baseline,  and  customized  DECtalk  and  the  Eloquent  system  as  test 
cases. 
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6.  ABBREVIATIONS  AND  ACRONYMS. 


AXIS 

Automatic  Terminal  Information  Service 

ATC 

Air  Traffic  Control 

FAA 

Federal  Aviation  Administration 

GAT 

General  Aviation  Trainer 

ILS 

Instrument  Landing  System 

NOT  AM 

Notice  to  Airmen 

VFR 

Visual  Flight  Rules 

VOR 

VHF  Omnirange 
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APPENDIX  A 


FLIGHT  ROUTES 


APPENDIX  B 


ATIS  MESSAGES 


Scenario  1 


Message  #  1  (Charlie) 

Atlantic  City  International  Airport  Information  Charlie.  One  four  zero  zero  Zulu. 
Measured  ceiling  four  thousand,  five  hundred  broken.  Visibility  one-and-one-half  fog. 
Temperature  four  niner.  Dew  point  four  seven.  Wind  one  five  zero  at  five.  Altimeter 
three  zero  zero  zero.  ILS  runway  one  three  approach  in  use.  Notices  to  airmen.  'Die 
last  one  thousand,  two  hundred  feet  of  runway  one  three  closed.  Eight  thousand,  eight 
hundred  feet  usable.  Taxiway  Bravo  between  taxiways  Juliet  and  Kilo  closed.  Runway 
three  one  VASIs  out  of  service.  Advise  on  initial  contact  you  have  information  Charlie. 

Message  #  2  (Delta) 

Atlantic  Qty  International  Airport  Information  Delta.  One  four  two  zero  Zulu. 
Measured  ceiling  four  thousand,  five  hundred  broken.  Visibility  one-and-one-half  fog. 
Temperature  five  zero.  Dew  point  four  eight.  Wind  two  seven  zero  at  five.  Altimeter 
two  niner  niner  two.  VOR  runway  three  one  approach  in  use.  Notices  to  airmen. 
Runway  three  one  threshold  displaced  one  thousand,  two  hundred  feet.  Eight  thousand, 
eight  hundred  feet  usable.  Relocated  threshold  identified  by  five-foot  solid  white  line 
and  runway  end  identifier  lights.  Taxiway  Bravo  between  taxiways  Juliet  and  Kilo  closed. 
Runway  three  one  VASIs  out  of  service.  Advise  on  initial  contact  you  have  information 

Delta. 

Scenario  2 

Message  #  1  (Echo) 

Atlantic  City  International  Airport  Information  Echo.  One  five  zero  zero  Zulu.  Sky 
partially  obscured.  Visibility  one-and-one-half  fog.  Temperature  four  five.  Dew  point 
four  three.  Wind  calm.  Altimeter  two  niner  niner  two.  VOR  runway  three  one 
approach  in  use.  Notices  to  airmen:  Runway  three  one  threshold  displaced  one 
thousand,  two  hundred  feet.  Eight  thousand,  eight  hundred  feet  usable.  Relocated 
threshold  identified  by  five  foot  solid  white  line  and  runway  end  identifier  lights. 

Taxiway  Bravo  between  Taxiways  Juliet  and  Kilo  closed.  Runway  three  one  VASIs  out 
of  service.  Advise  on  initial  contact  you  have  information  Echo. 
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Message  #  2  (Foxtrot) 

Atlantic  City  International  Airport  Information  Foxtrot.  One  five  one  five  Zulu.  Sky 
partially  obscured.  Visibility  one-half  fog.  Temperature  four  five.  Dew  point  four  four. 
Wind  one  five  zero  at  five.  Altimeter  three  zero  zero  zero.  ILS  runway  one  three 
approach  in  use.  Notices  to  airmen:  The  last  one  thousand,  two  hundred  feet  of  runway 
one  three  closed.  Eight  thousand,  eight  hundred  feet  usable.  Taxiways  Bravo  between 
Taxiways  Juliet  and  Kilo  closed.  Runway  three  one  VASIs  out  of  service.  Advise  on 
initial  contact  you  have  information  Foxtrot. 
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APPENDIX  C 


ORDER  OF  PRESENTATION 


Day  1:  Date:  November  28,  1994  (Monday) 

Morning  session  Pilot  1;  Scenario  1  (Human  Charlie,  Synthetic  Delta) 

Scenario  2  (Synthetic  Echo,  Human  Foxtrot) 

Afternoon  session  Pilot  2:  Scenario  2  (Human  Echo,  Synthetic  Foxtrot) 

Scenario  1  (Synthetic  Charlie,  Human  Delta) 

Day  2:  Date:  November  29,  1994  (Tuesday) 

Morning  session  Pilot  3:  Scenario  2  (Synthetic  Echo,  Human  Foxtrot) 

Scenario  1  (Human  Charlie,  Synthetic  Delta) 

Afternoon  session  Pilot  4:  Scenario  1  (Human  Charlie,  Synthetic  Delta) 

Scenario  2  (Synthetic  Echo,  Human  Foxtrot) 

Day  3:  Date:  November  30,  1994  (Wednesday) 

Morning  session  Pilot  5:  Scenario  1  (Human  Charlie,  S3mthetic  Delta) 

Scenario  2  (Synthetic  Echo,  Human  Foxtrot) 

Afternoon  session  Pilot  6:  Scenario  1  (Human  Charlie,  Synthetic  Delta) 

Scenario  2  (Synthetic  Echo,  Human  Foxtrot) 
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Day  4;  Date:  December  1,  1994  (Thursday) 

Morning  session  Pilot  7:  Scenario  2  (Synthetic  Echo,  Human  Foxtrot) 

Scenario  1  (Human  Charlie,  Synthetic  Delta) 

Afternoon  session  Pilot  8:  Scenario  2  (Human  Echo,  Synthetic  Foxtrot) 

Scenario  1  (Synthetic  Charlie,  Human  Delta) 

Day  5:  Date:  December  2,  1994  (Friday) 

Morning  session  Pilot  9:  Scenario  1  (Synthetic  Charlie,  Human  Delta) 

Scenario  2  (Human  Echo,  Synthetic  Foxtrot) 

Afternoon  session  Pilot  10:  Scenario  2  (Human  Echo,  Synthetic  Foxtrot) 

Scenario  1  (Synthetic  Charlie,  Human  Delta) 
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APPENDIX  D 


SCHEDULE 


Shakedown  Date:  November  14,  1994  (Monday) 

Test  Dates:  November  28  to  December  2,  1994 
Time;  8  a.m.  to  4  p.m. 

There  were  two  sessions  every  day.  Each  session  will  have  two  flights.  Each 
flight  took  approximately  40  minutes. 

Training  Period:  8  a.m.  to  9  a.m. 

Morning  Session;  9  a.m.  to  11  a.m. 

Lunch:  11  a.m.  to  12  p.m. 

Training  Period:  12  p.m.  to  1  p.m. 

Afternoon  Session:  1  p.m.  to  3  p.m. 

Demonstration/Debrief:  3  p.m.  to  4  p.m. 
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APPENDIX  E 


OBSERVER  QUESTIONNAIRE 


OBSERVER  QUESTIONNAIRE 
SIMULATION:  SYNTHETIC  VOICE  FOR  ATIS  MESSAGES 


Name: 

Hours  of  multi-engine  experience: 

Date:  Session:  Morning  /  Afternoon  (circle  one) 

As  an  observer,  you  will  ask  the  pilot  the  altimeter  reading  and  runway  after  the  ATIS  message  is 
complete.  There  will  be  two  ATIS  messages  in  each  scenario.  The  two  will  differ  in  their 
message  content  and  their  source.  Each  pilot  will  complete  two  scenarios’.' 


SCENARIO  1 


ATIS  Message:  Charlie 

Nature  of  message:  Human  /  Synthetic  (circle  one) 

1 .  What  is  the  pilot's  response  to  your  question  "what  was  the  aldmeter  reading"? 


2.  What  is  the  pilot's  response  to  your  question  "what  runway  is  in  use  now"? 


3 .  How  many  times  did  the  pilot  listen  to  the  message? 


ATIS  Message:  Delta 

Nature  of  message:  Human  /  Synthetic  (circle  one) 


1. 

What  is  the  pilot's  response  to  your  question  "what  was  the  altimeter  reading"? 

2. 

What  is  the  pilot's  response  to  your  question  "what  runway  is  in  use  now"? 

3. 

How  many  times  did  the  pilot  listen  to  the  message? 
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SCENARIO  2 


ATIS  Message:  Echo 

Nature  of  message;  Human  /  Synthetic  (circle  one) 

1.  What  is  the  pilot's  response  to  your  question  "what  was  the  altimeter  reading"? 

2.  What  is  the  pilot's  response  to  your  question  "what  runway  is  in  use  now"? 

3 .  How  many  times  did  the  pilot  listen  to  the  message? 


ATIS  Message:  Foxtrot 

Nature  of  message:  Human  /  Synthetic  (circle  one) 

1.  What  is  the  pilot's  response  to  your  question  "what  was  the  altimeter  reading"? 

2.  What  is  the  pilot's  response  to  your  question  "what  runway  is  in  use  now"? 

3 .  How  many  times  did  the  pilot  listen  to  the  message? 


General  Questions 

1 .  What  is  your  overall  opinion  about  the  synthetic  ATIS  message? 

Acceptable  /  Unacceptable  (circle  one) 

2.  Do  you  have  any  additional  comments  or  opinions  about  the  synthetic  AUS  messages  (use 
back  side  if  necessary)? 


Please  return  to  Parimal  Kopardekar.  Thank  you  for  vour  participation  in  the  study. 


E-2 


APPENDIX  F 


PILOT  QUESTIONNAIRE 


PILOT  QUESTIONNAIRE 

SIMULATION:  SYNTHETIC  VOICE  FOR  ATIS  MESSAGES 

Name: 

Hours  of  multi-engine  experience:  Date: 

Quality  of  synthetic  voice 

1 .  How  did  the  speed  of  the  synthetic  voice  sound  to  you? 

1  2  3  4  5  6  7 

Too  Fast  Adequate  Too  Slow 

Please  justify  your  response  if  it  is  any  other  than  4  (use  back  side  if  necessary). 


2 .  Do  you  think  the  synthetic  ATIS  message  was  clear  and  intelligible? 

12345  6  7 

Not  Clear  Somewhat  Completely  Clear 

Clear 

Please  justify  your  response  if  it  is  any  other  than  7  (use  back  side  if  necessary). 


3 .  Which  type  of  ATIS  message  do  you  prefer?  (circle  one) 

Natural  Voice  Synthetic  Voice  No  Preference 

4 .  Do  you  think  the  overall  quality  of  ATIS  message,  recorded  in  synthetic  voice,  is 
acceptable? 

1  2  3  4  5  6  7 

Not  At  All  Completely 

Please  justify  your  response  if  it  is  any  other  than  7  (use  back  side  if  necessary). 


5 .  Please  feel  to  mention  any  other  comments,  opinions,  or  concerns  you  may  have  about 
synthetic  ATIS  message  (use  back  side). 


Please  renim  to  Parimal  Konardekar,  Thank  you  for  vour  participation  in  the  survey,. 
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APPENDIX  G 


COMMENTS  FROM  PILOT  AND  OBSERVER  QUESTIONNAIRES 


PILOT  RATINGS  AND  COMMENTS 

NOTE:  On  questions  where  pilots  were  asked  to  provide  both  a  rating  and  comments, 
the  rating  is  shown  before  the  comments.  Refer  to  Appendix  F  for  a  description  of  the 
rating  scales  used  for  each  question. 

1.  How  did  the  speed  of  the  synthetic  voice  sound  to  you? 

Pilot  1:  6  —  Information  dragged  out  too  much,  distracted  me  from  flying. 

Pilot  2:  4  —  No  comment. 

Pilot  3:  4  —  No  comment. 

Pilot  4:  4  —  No  comment. 

Pilot  5:  5  —  Voice  too  slow. 

Pilot  6:  4  —  No  comment. 

Pilot  7:  5  —  Seemed  a  little  slower  than  a  normal  ATIS,  but  not  really  objectionable. 

Pilot  8:  4  —  No  comment. 

Pilot  9:  4  —  No  comment. 

Pilot  10:  4  —  No  comment. 

2.  Do  you  think  the  synthetic  ATIS  message  was  clear  and  intelligible? 

Pilot  1:  4  —  Certain  words  yes,  others  were  not. 

Pilot  2:  7  —  No  comment. 

Pilot  3:  4  —  Not  as  clear  as  the  GAI AWOS  I  am  used  to  listening  to  when  I  fly  there. 

It  is  also  not  a  real  human  voice. 

Pilot  4:  7  —  No  problem  understanding  the  verbiage. 

Pilot  5:  7  —  No  comment. 

Pilot  6:  4  —  Because  the  voice  was  of  foreign  origin,  more  concentration  is  required  to 

understand  words. 
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Pilot  7:  6  —  Had  trouble  once  understanding  the  number  “five’;  also  seemed  to  have  a 

slight  foreign  accent  (maybe  Scandinavian). 

Pilot  8:  6  —  The  receiver/speaker  was  little  touchy,  but  the  voice  message  was  fine. 

Pilot  9:  4  —  Some  words  difficult  to  understand— hard  consonance  in  particular.  Had  to 

listen  twice  to  ATIS  to  get  some  words. 

Pilot  10:  7  —  No  comment. 

3.  Which  type  of  ATIS  message  do  you  prefer? 


Pilot  1: 

Natural  voice. 

Pilot  2: 

Natural  voice. 

Pilot  3: 

Natural  voice, 

Pilot  4: 

No  preference. 

Pilot  5: 

Natural  voice. 

Pilot  6: 

Natural  voice. 

Pilot  7: 

No  preference. 

Pilot  8: 

Synthetic  voice. 

Pilot  9: 

Natural  voice. 

Pilot  10: 

Synthetic  voice. 
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4.  Do  you  think  the  overall  quality  of  ATIS  message,  recorded  in  synthetic  voice,  is 

acceptable? 

Pilot  1:  2  —  I  am  currently  only  familiar  with  natural  voice.  In  high-workload 

situations,  synthetic  voice  was  too  difficult  to  interpret. 

Pilot  2;  6  —  It  will  require  some  time  to  get  used  to  for  the  uninitiated. 

Pilot  3:  4  —  Could  be  clearer.  I  really  don’t  have  any  preference  of  human  voice 

versus  synthetic,  as  long  as  it  is  clear. 

Pilot  4:  7  —  No  comment. 

Pilot  5:  5  —  Voice  was  too  slow. 

Pilot  6:  4  —  Because  the  voice  was  of  foreign  origin,  more  concentration  is  required  to 

understand  words. 

Pilot  7:  7  —  Could  be  slightly  better  (a  little  faster),  but  [is]  at  least  as  good  as  the 

average  of  all  natural  voice  ATIS  I’ve  heard. 

Pilot  8:  6  —  Very  good  voice  and  pattern.  I  had  a  little  problem,  but  it  was  only 

unfamiliarity  with  the  trainer. 

Pilot  9:  4  —  Somewhat  difficult  to  understand  at  times;  required  more  attention  and 

concentration  compared  to  natural  voice.  On  second  ATC  scenario  did  not 
attempt  to  listen  to  ATIS  until  aircraft  was  in  benign  phase  of  flight  (i.e., 
stable  heading  and  altitude). 

Pilot  10:  7  —  No  comment. 

5.  Please  feel  free  to  mention  any  other  comments,  opinions,  or  concerns  you  may  have 

about  synthetic  ATIS  message. 

Pilot  1:  It  was  too  hard  to  interpret  synthetic  voice  ATIS  in  a  high-workload  situation. 
It  distracted  me  from  flying  because  I  had  to  concentrate  on  what  was  being 
said.  In  a  lower  work  situation  during  the  test  I  understood  it  better. 

Pilot  2:  No  comment. 

Pilot  3:  No  comment. 
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Pilot  4:  For  my  level  of  experience  (low  multiengine  time),  I  was  quite  busy.  As  long 
as  the  synthetic  ATIS  is  clear  with  the  same  order  of  information  that  the  pilot 
is  used  to,  it  is  adequate. 

Pilot  5:  No  comment. 

Pilot  6:  No  comment. 

Pilot  7:  I  suggest  you  get  some  subject  pilots  who  are  not  native  English  speakers;  they 
would  be  more  sensitive  to  this  than  I  am.  The  NOTAMs  in  this  ATIS  were 
exactly  what  we’re  used  to  now  here  at  ACY.  It  might  be  better  to  put  in 
some  nonstandard  NOTAMs. 

Pilot  8:  I  feel  it  is  better  than  human  because  of  the  consistency. 

Pilot  9:  Synthetic  voice  ATIS  needs  improvement  in  voice  quality  before 

implementation.  Need  to  approach  quality  of  natural  voice  to  minimize 
concentration  required  to  comprehend  all  data. 

Pilot  10:  No  comment. 

6.  How  do  you  find  the  background  traffic? 

Pilot  1:  No  response. 

Pilot  2:  Normal. 

Pilot  3:  Normal. 

Pilot  4:  Light  side  of  normal. 

Pilot  5:  Normal. 

Pilot  6:  Normal  to  light. 

Pilot  7:  Normal  (normal  for  ACY  but  light  compared  with  a  lot  of  other 
environments). 

Pilot  8:  Normal. 

Pilot  9:  Normal. 


Pilot  10:  Normal. 


OBSERVER  COMMENTS 


The  following  comments  were  made  by  the  pilots  to  the  observer  during  the  flight  and 
were  recorded  by  the  observer  in  response  to  general  question  2  on  the  observer 
questionnaire  (refer  to  Appendix  E). 

Do  you  have  any  additional  comments  or  opinions  about  the  synthetic  ATIS  messages? 

Pilot  1;  Multisyllabic  words  are  hard  to  understand— pause  too  great  (i.e.,  visibility  1 
mile  or  1  and  1/2)?  Pilot  flies  for  a  living. 

Pilot  2:  Length  of  AXIS  NOTAMs  has  some  bearing  on  number  of  times  a  pilot  will 
listen  to  the  ATIS. 

Brief  pilots  on  glide  slope  flag  in  view  on  ILS. 

NOTAM  messages  are  more  difficult  to  understand.  Subjects  may  get 
altimeter  and  runway  but  not  understand  NOTAMs  with  synthetic. 

As  far  as  comparing  synthetic  to  human,  I’ve  heard  much  better  human  ATIS 
and  a  few  that  were  worse.  Perhaps  consider  saving  human  voice  words  on 
tapes  and  use  them. 

Pilot  3:  Pilot  workload  was  high,  not  experienced  with  this  type  flight  deck.  ATIS 
[(syn)synthetic]  was  just  acceptable,  required  more  attention. 

Pilot  4:  Pilot  did  not  use  autopilot. 

Pilot  5:  Pilot  went  back  approximately  7-10  min  after  start  of  scenario  2  and  listened 
to  AUS  echo  (only  about  half). 

Pilot  6:  The  pilot  during  this  problem  appeared  to  be  “behind  the  power  curve” 

complying  with  ATC  instructions.  When  the  ATIS  messages  changed  from 
Charlie  to  Delta,  the  pilot  immediately  went  to  receive  the  new  information 
and  did  not  hear  (or  comply  with)  instructions  which  would  allow  the  aircraft 
to  proceed  as  scripted. 

Pilot  7:  Pilot  used  autopilot  during  high-workload  times. 

Pilot  8:  Used  autopilot. 

Pilot  9:  Pilot  used  autopilot  all  the  time.  Pilot  flies  for  a  living. 

Pilot  10;  No  comment. 
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APPENDIX  H 


PERSONNEL 


Test  Director — ^Albert  Rehmann 


Scenario  Development  and  ATC  Support — Frank  Ferrera 

AXIS  Messages  Preparation — ^John  Goon  and  Parimal  Kopardekar  (CTA  Incorporated) 

ATC  Support — Evan  Darby 

General  Aviation  Simulation  Facility  Engineering  Support — Joe  McCall  and  George 
Bollenbach 

Expert  Pilot  and  Pilot  Scheduling — Theos  McKinney 

Software  Support — Joe  McCall,  George  Bollenbach,  and  Pocholo  Bravo 

Test  Plan  and  Study  Preparation — Parimal  Kopardekar  (CTA  Incorporated),  Richard  Mogford 
(CTA  Incorporated),  and  Margaret  T.  Shaffer  (ARINC) 

Data  Collection  and  General  Support — Parimal  Kopardekar  (CTA  Incorporated),  Margaet  T. 
Shaffer  (ARINC) 

Data  Analysis  and  Report  Preparation — ^Margaret  T.  Shaffer  (ARINC) 
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